is happily being the wheel rather than a rusty old spoke
Register for free to join our community of investors and share your ideas. You will also get access to streaming quotes, interactive charts, trades, portfolio, live options flow and more tools.
Register for free to join our community of investors and share your ideas. You will also get access to streaming quotes, interactive charts, trades, portfolio, live options flow and more tools.
I'll rarely, if ever, say when I'm going to get something done. If I say a feature will be done by such and such a date, I'm probably lying to you.
However, I might let you know when it's done.
Or I might not say anything and let you discover on your own that it's done. <g>
This is an excellent idea! Once you guys hash out a weighting algorithm and a list of stocks, I'd be more than happy to put together a database and code to track it. Unless Matt's already pretty far along on it.
I just checked and am finding nothing on this end to support your conclusion.
Do you have something you can offer that does? I'll be happy to check it out. I'm very averse to people having multiple accounts.
I see two possible answers:
1. I screwed up. Again!
2. Some people who had you peoplemarked hadn't posted for a while until today. Each of those people would've caused a jump of 2 points in your score.
Checking on you.
Okay, I just calculated yours by hand and it should be 30 points. 13 of those points would've come from posting volume of 410 posts in the last 30 days. Does that figure sound about right?
I should add that if the concern is that someone will move themselves up the list by posting a bunch of BS (as a quick perusal of that profile confirmed), that problem has another solution, which should be implemented shortly.
I'm not sure I understand what you're getting at. It's not like we're putting dollar signs next to peoples' names.
To get 32 points based only on posts, one would have to write 960 posts in 30 days. And I haven't seen any kind of unusual number of new memberships.
No matter what I do with that formula, it'll be possible to abuse it. I doubt it'll be possible to abuse it so much that someone gets themselves into the top 10, though. And as the site gets busier, it'll be tougher to even get *on* the list, let alone manipulate your way to the top. Especially since Matt's all over the multi-ID thing. It's important to both of us that people don't use multiple accounts here.
It's seeming, too, that no matter what I do to the formula, there will always be 3 factions:
1. Do away with it! We don't want such a list to exist!
2. It's not measuring what should be measured.
3. Don't do away with it. Such a list should exist!
One of the things that made me think to include it here is what a big deal it was on SI and the countless messages I got from people asking when the next one would be out.
I still maintain that it's a useful tool for people coming to the site for the first time. Without it, they've got no way of seeing where the action is. For all they know, it could be 5 people posting to themselves or it could be 5 million.
I'm also not convinced that there's anything so urgently "wrong" with it that changes to it can't wait a little while so I can do other things and so more people can offer their opinions. Especially since that list isn't such a burning issue to me beyond it accomplishing the main goal of easily letting people see who's here and what they're posting.
Knew it couldn't only be in read_msg. Those contention problems are going to pop up a lot of places because of the way locking is handled.
A possible interim solution until I'm ready to tackle the locking issue (at the same time I move stuff to stored procs) would be to throttle down IIS to reduce the number of concurrent requests the SQL box is having to handle. The IIS box is spitting out commands to the SQL box real rapidly.
Looking at some of the top 50, I'd say it meets that goal. Wouldn't you?
There's always going to be some downside. I think the whole idea works out net positive. But the more net I can get out of it, the better. I'm just not going to throw the whole thing out because it's not 100% upside.
Figured that one *had* to be Dylan. Point well taken and constantly in the back of my mind.
My main intention for the lists (members and boards) is for it to be a getting-started point for people who've never been here before.
A secondary, but still existent reason for it is the ego-stroking aspect.
I expect the primary users to be new members. Secondary users will be established members who care about such things.
I agree to a certain extent. I think it's helpful for someone who's never been here before to go see who's active and who's widely read. Beyond that, I would expect most would decide for themselves just who they're going to read.
Some of the people I read on SI I feel are "best-kept secrets". They wouldn't show up on such a list until more people discovered them. Doesn't make them any less valuable to me.
So, the list is relevant and irrelevant at the same time. Depends on who's using it.
Well, as the site grows (which it seems to be doing daily), those numbers should grow, too. But I'm kinda thinking about multiplying PM and APM to give APD less weighting.
Jenna's score is lower than yours because she posts less and a higher percentage of the people who have you marked have also posted in the last 30 days. Once I'm tracking last-visit-date, I'll use that instead of or in addition to the APM figure.
As it stands now, the most glaring "incorrectness" is that I would expect that most of her audience has her thread bookmarked rather than having her personally bookmarked. I should probably factor this in, too, sometime, but even that is rife with "problems".
Then how did they hire you? Did you just tell them you know what your doing?
I'm mostly self-taught. Yes, like in any line of work, the first gig was tough to get, but once I'd proven myself on that one, the ones that came later were much easier to get.
Could I go to a construction company or whatever and tell them that I can drive one of those tractor things - and they'd let me??
No, but I probably could, even though it's not on my resume. I would just tell them I've got one of those "tractor things" and know how to use it, they'd likely test me to see if I really could, then put me to work.
Worked the same with programming. Had to pass the initial testing to make sure I wasn't BS'ing them. Then keep the job by showing I really knew what I was doing.
Sure, there's also a "BS Factor". A running joke between my best friend and I is "That's what I do best!". It's the answer I learned to give to prospective consulting clients. Even if they asked if I knew a particular language that I didn't really know. You get enough languages under your belt and adopting a new one is just a matter of having a quick-reference handy so you know little things like whether it's "endif" or "end if".
The thing is, the executives ask that; the IT managers (if they're former programmers) typically don't. Programmers know that the really necessary skill is the understanding of programming conceptually.
As it turns out, if I'm interviewing for the role of programmer here, this software's written in a dialect of the first programming language I learned. This is my first real heavy work in ASP, but VBScript is like English to me. With a southern accent.
To clear up a misunderstanding you betrayed in your first post to me about it, ASP isn't a programming language. It stands for Active Server Pages. It's more concept than language. Code to take user input, process it, and produce HTML output for them. The code can be in just about any language. VBScript seems to be the language of choice these days, along with JavaScript. I think the current version of SI relies heavily on Java, but if I remember correctly, the Dryer version of SI was written in C.
In any event, I've given lots of explanation (just 'cuz it's fun), but I'm not sure what your point is. That I couldn't possibly be a programmer? If so, I guess my answer is that it's "what I do best". <g>
What the hell is a road course?
A flavor of race course. My favorite flavor. Unlike ovals, you deal with lots of turns. I think the one in Omaha is going to have something like 14 turns on it plus some generous straightaways. It's not as big as Heartland Park (Topeka), so I'm guessing I'll be pulling a max of about 115mph in my "playing around car" and about 135 in the racecar. It has lots of tricky turns but looks like it'll be easier on brakes and tires than Heartland.
If you like roller-coasters, you'd love riding along.
If you can really *drive*, they're great fun. If you can't, it's a good way to learn.
I love cars,
Well, then we do have some common ground. I love cars, motorcycles, tractors, trains, and anything with wheels and a motor.
A real shame about that Ferrari. I'd love to see more of those on the racetrack, but in all my years of doing the road-course stuff, I've only encountered one.
Seeing as how you live in Nebraska, don't you feel a little self-conscious when you use the word "hillbilly"?
I've re-implemented Top Members on the Hot List and am handling it completely differently. If you've got feedback on how the formula goes, I'm all ears, via PM or in the Q&A thread.
You'll notice that the scores are now completely different and so is the order. Here's why:
First, rather than assigning a point value to a person's posts depending on the number of peoplemarks they have, I simplified that to just 1 point per average posts per day for the past 30 days. Each of Joemoney's posts are just as "valuable" for scoring as each of Matt's. Everyone gets the credited points for posting volume on the same scale. It's not "correct" in my view, but it's a heckuva lot easier to deal with. The old way was likely the culprit in the "divide by zero" errors.
In addition to counting a person's total peoplemarks, I also count the peoplemarks they have from posters who've posted at least once in the past 30 days. Those peoplemarks count triple. The previous version didn't take these into account. This, I think, is a lot more "correct". That way we give less value to the peoplemarks of people who don't actively use the site.
A much simpler formula (though not easy at all from a SQL viewpoint), but we'll probably complicate it a bit later.
Kinda/sorta.
I went through a lot of queries that really didn't need to fetch entire tables and made them fetch TOP 50 or whatever amount worked, instead.
read_msg will be a complete rewrite.
Since it's the remaining major culprit, I expect to address a lot of different things with the rewrite: Stored Procs, forcing SQL to handle locks my way (using NOLOCKS where warranted), and using more-selective SELECTs. <g>
I'm not in a big hurry, though. I think enough of it's tweaked now to make the timeouts a less frequent occurrence, though likely not rare yet. I'm going to clear the rest of the stuff off my plate then dive into read_msg. Once I'm done with read_msg, I should be done with performance tweaks for a while and can knock out some bugs and new features before going back to the optimization thing. You know, mostly CSS and stored procs across the site.
You're really a lot of fun sometimes, Joe.
I would communicate with Matt through email or PMs instead of publicizing every update you make.
I disagree. I do communicate with him regularly, but I like the idea of a running history of updates made to the site. Besides, in a lot of cases, people would have no idea a new feature existed if I didn't say so. Like the post I'm going to make to the Bowie thread after this post.
And if you've been doing programming for 12 years and did it for a living, then you must have gone to school to learn that.
Irrelevant and wrong.
And if that's true, then I believe that you know something about what your doing.
Again, irrelevant and wrong. But since you're apparently not qualified to determine whether or not I know what I'm doing, it's also moot.
And operate a backhoe? How hard can it be?
Operating one skillfully, efficiently, and safely is quite a feat. I consider myself good at it, but not great. I didn't go to school for it. However, I have a friend who's a GREAT backhoe operator and he didn't go to school for it either.
If you can drive a car, I'm sure I can operate a back hoe.
Well, I'm sure that you could "drive" a backhoe. But not really put it through its paces. It's one thing to put it in gear and steer it down the road. Quite another to pick up an egg without breaking it. Kinda like the difference between the level at which I "drive" and the level at which you do. I'm, like, *real* good at that. hehe
In all seriousness, you guys are getting a new road-course in Omaha and I'll be running on it at least once next year. You're invited. Bring a helmet. I'll supply the barf bag. :)
However, I'm not good with manual transmission - I crashed my first car with manual transmission about 2 years ago, so as long Catapillar makes an automatic - how hard could it be?
The difference between operating a backhoe and driving a car is far greater than driving an automatic and driving a stick. When I refer to "operating" a backhoe, I'm not talking about driving it. I'm talking about things like digging trenches within inches of existing gas lines. Stuff like that. It really takes a while to get the hang of it because both feet and both hands are in constant motion. And each hand controlling a joystick that moves in any direction, with each direction controlling a separate function. It's second-nature for me, but it took probably 100 hours or more before it became that way. And I still screw up sometimes.
Besides, I'd rather hire someone.
Dude, you're apparently not a gear-head. These things are a blast!!!
Are you one of those people that considers a car just a way to get from one place to another?
common sense says - "Don't get your hands dirty, that's their job."
That's not common sense. That's personal preference. Sure, there are plenty of things I could do myself that I'd rather write a check for (hanging/mudding sheetrock is at the top of that list), but playing with big Tonka Toys sure ain't one of 'em.
I'm serious about that invite to the track in Omaha. Goes out to anyone here. Since it'll be my first time on that track, I doubt I'll be going fast enough to really scare anyone. I'll letcha know when I know what date(s) I'll be up there.
Nah. Nothing on your end. Only two people should ever have an "I'm editing this record" kind of lock on a post: The author anytime they do an update on it and the recipient anytime they read it. Did you edit the post at all? I suppose it's possible that you were posting an edit at precisely the same fraction of a second that X hit the link that went to that post, but I seriously doubt it.
Good. That tells me about what time it happened. I'd imagine during that time the site was super-slow for everyone and there were a number of timeouts.
That wouldn't be too hard to implement. What do you think of factoring those (lightly) into someone's "score"? As long as we stay a million miles away from the way RB would "score" people.
Let me guess. My trace log says we all had some problems this morning, eh? Message 246747 caused a lot of problems this morning. And attempts to read it were tying up the server so badly, there were timeouts galore! Glad I traced the site. It's real obvious what happened now. Just gotta figure out how to fix it.
Definitely a locking issue. When the recipient tried to read it at the same time that someone else was trying to read it, all hell broke loose. Because when the recipient reads a message, a field in that record gets changed. One connection trying to read (and apparently lock at the same time), while another connection is definitely trying to lock it so it can update it.
I'm on it.
Can't believe I missed this one earlier...
I didn't know
Now there's a hole with no bottom
he knew a damn thing about computers, let alone ASP.
Does your not knowing something make it not true?
BTW, if you knew anything about computers, you wouldn't have put in that "let alone ASP". That's kind of like saying I didn't know you could understand English, let alone English spoken with a mid-western accent.
Don't you know that one of the hallmarks of genius is expertise in a number of dissimilar areas? <g> Would you be similarly incredulous if I mentioned I'm an accomplished musician, a helluva deck builder, and am a good Daddy to boot?
I post about the coding changes to keep people abreast of what's going on. Pretty simple.
Oh, and I was a computer consultant (programmer/database analyst/business process analyst) for about a dozen years before I went to work at Silicon Investor. Geeking's not exactly a new thing for me.
So, I'm really kinda curious. Just *what* is it you're trying to say? That I don't know anything about ASP and haven't really done anything? Or if your point is that I shouldn't say anything about it, why is that? Because it bugs you personally? Can you show me some downside to that?
I think it's important to let people know about the changes so when my changes screw things up (which they will eventually -- you can only evade Murphy for so long), we'll know exactly what change did it.
But if on one hand you think I'm bragging and it's unseemly (I'll grant you that one -- I don't agree, but it's easier than arguing it), at least give me credit for being better than the SI staff who, instead, say "Oh, yes. We'll get right on that. You should have that feature any day now." and it drags on for months or the promised feature never materializes.
So, how would you do it better? Assuming you had the smarts necessary to both operate a backhoe and wade into a bunch of ASP.
http://www.investorshub.com/boards/read_msg.asp?message_id=247071
Not only should timeouts either go away or be much rarer, the whole site should run a LOT faster now.
Lots of changes tonight, most of which shouldn't be noticeable in terms of what the site looks like. At least, I HOPE not!
I've been getting reports of timeouts, specifically in the routine that marks a public message as being read once you go to it. That routine itself shouldn't be too expensive, so I suspect it might be a timing and locking issue.
In researching it, I found that there are a LOT of expensive queries. I put a trace on them and had it show me every query that took longer than 1000 duration (I'm not convinced it's really milliseconds).
By getting rid of some indexes, adding others, and changing the way a lot of queries are handled, I decided to run the trace again. 15 minutes with nothing more expensive than 1000. So I had it start showing everything that cost more than 500. That turned up quite a few more, so I did a bunch of tweaking here and there, and started the trace again.
That was about 15 minutes ago and there's still been nothing more expensive than 500 showing up.
So, let's see how it goes.
Watch for weird things in the following areas and let me know if you see anything awry:
1. List of messages within a board.
2. List of messages within a profile.
3. Your Mail Box.
Note that one change I did make that should be immediately noticeable is that when you go to your mailbox, only the 50 most recent private and 50 most recent public messages to you will display. I hope this isn't too inconvenient. I'll add something later where you can view ALL of your messages but leave the most recent 50 as the default view.
Viewing all of your messages was proving to be the most expensive process on the whole site. If anyone had more than a few hundred messages saved, it was killing us every time they'd go to their inbox.
Keep in mind the messages still exist and their status (not read) hasn't changed. The only change is that only the most recent 100 messages (50 public; 50 private) are shown.
Also, I've noticed that CPU usage on the SQL box has plummeted. Though I can see that there are still quite a few users active on the site right now.
Because it's a very temporary fix. I was in the middle of something else (and still am), so I just made a quick change in the query and left it at that.
I was pretty sure I posted a message in this thread about the change.
Stop-gap while I get done on current stuff, then I'll go see what on earth I did wrong in that formula.
Now only Peoplemarks are used to calculate the list.
To kind of get the ball rolling, here's my current struggle. I think I've got a handle on it, but any ASP/SQL geeks out there are more than welcome to offer their opinion.
The site has been getting frequent timeouts. Specifically, web pages that don't load and instead give the "timeout expired" message. I haven't gotten enough cut and paste copies to know for certain, but I think the main culprit might be a particular SELECT statement in read_msg.asp. However, the trace that's currently running might be identifying boards.asp as the biggest culprit. Won't know for sure until I scrutinize the trace log tonight.
Both the SELECT statement I've seen bomb out via cut and paste, and the one that's being identified as a resource pig in Trace right now have at least one thing in common (that I've noticed so far): the SELECT statement specifically naming every field to retrieve (over a dozen) rather than using a "*". Oddly, when I tried changing it to "*" on my local setup, I got errors further down the script. Go figure.
I doubt the explicit naming of fields is really the problem anyway. It's not the angle I'm going to pursue first to try to tweak it.
I think the problem might instead be the WHERE clauses. I haven't checked yet (I will tonight), but it's entirely possible that the fields named in the WHERE clauses aren't indexed.
One of the fields can contain one of only two possible values and another can contain only one of 3. It's my understanding that you don't typically index such fields. Thoughts?
The field that contains 3 possibilities is used to identify what type of message is being looked at and to exclude private messages. I've got an idea on that one. It's looking for one value to exist in that field (which is a Text field, btw -- I'm thinking that could be a big problem) and I'm thinking that I should add a field (type Bit) to the table that's used to identify the message as either private or not.
I'll likely try that tonight as well as making it a Stored Proc to see if it can work a bit faster.
So, while I have my best friend (who's a major SQL Server geek) look at the server configuration, I'll try to tweak all queries so that none of them take more than 1k milliseconds to run. The first approach I'll use is trying to get them to do the fewest reads possible.
Likely related to the timeout problem. I hadn't caught it before because the refresh rate at which I was watching CPU usage was every 10 seconds. So if the CPU was 100% busy for 4 of those seconds and only 20% busy the rest of the time, it'd show up as 52% utilization. I thought that was a bit high but not bad enough that it needed quick attention. Now I know differently.
What I know so far is that one particular line of board.asp, when applied to a board with a lot of posts (like "Five and Under", which has over 28k posts) is taking well over a second to execute and does over 5k reads in the process.
Once I'm certain it's that particular query (as the trace should confirm tonight), I'll make it a Stored Procedure and tweak it to do as few reads as possible.
Timeout issue
Likely a SQL Server configuration issue. Researching it. Or it could have something to do with the amount of traffic finally reaching a point that the way queries are handled has become a problem.
When John gets an opportunity, I'll ask him to go through the server with a fine-toothed comb to make sure it's configured as it should be.
I started a trace this morning that'll run until midnight tonight and is logging every transaction that takes longer than a second to complete. Been running about 5 minutes and has only logged 50 of them so far. But IMO *nothing* should take longer than a second to do. And 1 query every 6 seconds that takes longer than a second (more like 1.5 seconds) to complete is too close to the edge.
I think the main culprit is board.asp. It has a number of SELECT statements in it, and one of the ones in it seems to be what's always taking a lot of time to execute. And doing a too-large number of reads.
So, when I shut off the trace tonight, I'll figure out which queries are taking up the most resources and rewrite them to be more efficient and see if that doesn't make the problem go away.
Profiles - editing them - i think it's the '/" thing
Too-low priority for me to work on it today.
Finishing off new subscription stuff
That's what I'm currently working on.
Aren't you grandfathered in?
I'm working on the registration/subscription pages and underlying processes. I've noticed we *do* give quite a bit away, though.
Or are you just pimping Matt?
The problem is definitely on this end. (Aside: When's the last time you heard a message board operator fess up to *that* one. LOL).
We have a number of processor-intensive queries running, but I hadn't noticed the processor getting all that tied up. My sampling rate might need to be adjusted to where I see ever 5 seconds of data, although that carries its own overhead.
Everyone, if you get the timeout, please cut and past the exact message. I don't think it's the same query doing it all the time. I think the issue is with SQL Server itself more than it is any particular query.
The way the queries are done (built on the ASP side then submitted) is the least-efficient way to do it (stored procedures would be much better), but they're not so inefficient that we should be having these kinds of problems. The SQL box usually looks like it's yawning.
It's likely a SQL/ASP problem. Dang! I'm finding recent reports of it all over the internet and finding no solutions yet. People are getting it when running the simplest SELECT statements even on tables as small as 100k rows.
I'll look tomorrow or Monday. A side thing that happened while I was trying to solve this problem really cost me my appetite for geeking tonight.
Too bad. Some hard-core geeking would've capped off what had been up to now a gloriously gratifying day. <g> Got a major Tonka Toy fix, all day! :) Sprained heck out of my ankle jumping out of the dumptruck once (picture me laying on the ground next to a running dumptruck bellowing in pain for minutes -- I jumped out to see why the PTO wasn't working -- had sheared a bolt) but even that didn't dampen my day. At least it wasn't my clutch ankle.
Looks like I'm going to need to get on it, if it's still happenning. When someone reported the problem last night with the whole error message, it was the line that marks a message as having been read. Real reluctant to dink with that bit of code, as I haven't been through it to learn it yet, but it seems to me it's got an unnecessary WHERE clause on it.
But I'll go through it and make sure I'm somewhat comfortable removing the WHERE clause and go ahead and do so and see what happens.
Come to think of it, I'll try it here first.
Edit: Wow! Had no idea that one was so expensive! I pegged my CPU on this end by hitting Reload on it a bunch.
"So help me, I'm gonna stop this car..."
As an aside, Joe, the TOS thing should work now. Mail services didn't come up after the reboot last night.
Thanks for giving the exact error message. That not only helps, it makes it actually *possible* to track down a problem.
Whew! I was worried it was yet another failure to connect to the database. Really shouldn't have any intranet problems since it's running on good INTC NIC cards.
It's a mail-server problem, Matt. The line being referred to says "Msg.Send"
I really thought I'd bumped that timeout up to an hour a week or so ago. It's been doing it to you recently?
The sure fix for that is to go ahead and accept the cookie and if you don't like carrying them around, clean 'em out every night when you're done. With cookies, there's no time-out issue as far as being expected to log back in.
The other possible solution would be for me to add another IIS front-end machine of equal power, and go through the rigors of load-balancing and making sure each person stays on the same IIS box until they log out. That's the route I'd have to take if I want to make the timeout some really hairy amount like 24 hours. Before long it'd have several thousand connections open that aren't "really" being used.
Same here. Haven't gotten any timeouts at all. Matt said he was getting them last night until I rebooted the machines, and he's on cable or something like that.
If the problem persists, I'll do a stop-gap measure of increasing timeouts. Optimizing everything for maximum speed and efficiency is still about 4 items down on the list. The current project (overhauling the registration/subscription process) is still 3-4 days from completion, and I'm not working on it at all until tonight.
Having too much fun outside with the backhoe and dumptruck. Basically moving dirt from one place to another. Good to spend some time doing something so analog.
Next time it happens could you give me details, like the time it happened and a cut and paste of what the screen said?
We're in agreement. Despite my having booted him numerous times (including a termination that was overridden by a community vote), I have the deepest respect for him.
As far as his past, yes, it amuses me when people act like it's some huge revelation and try to use that as a reason nobody should pay attention to him.
To use an analogy, who's better equipped to educate people about the evils of alcoholism: the lifelong tee-totaler or the reformed alcoholic?
I won't even *remotely* go so far as to say he's never been wrong. But, like with Michail Shadkin, if I'm in a stock and he posts something negative about it, I pay darned close attention and do some real DD to determine whether or not I agree with him.
Oh, and "colorful"? Man! There've been times I've written to him that though I'm laughing my butt off at something he wrote, he's also taking a vacation over it.
I consider him, in the aggregate, a good guy, based mainly on the number of people he has helped avoid or get out of scams or just overpriced stocks.